Arbiter (Halo 3 Reclaimer)

+ +

+ + +
Field Summary
+ + + + + + + + + + + + + + +
Fields
Modifier and Type Field and Description
static java.lang.String MAPS_DIR
static java.lang.String SCREENSHOTS_DIR
+

Fields
Modifier and Type	Field and Description
`static java.lang.String`	`MAPS_DIR`
`static java.lang.String`	`SCREENSHOTS_DIR`

@@ -157,12 +191,10 @@ extends java.lang.Object static java.lang.String -getFileTypeDescription(java.io.File inputFile) - - -static void -

processFile(java.io.File inputFile,
-           java.io.File outputDir)

processFile(java.io.File input,
+           java.io.File outputDir)

Reclaims all screenshots and maps found in a file or folder.

+ +

+ + +
Field Detail
+ + + +
- +
  SCREENSHOTS_DIR
  +
```
public static final java.lang.String SCREENSHOTS_DIR
```
  +
  +
  See Also:
  +
  Constant Field Values
  +
  +
+ + + +
- +
  MAPS_DIR
  +
```
public static final java.lang.String MAPS_DIR
```
  +
  +
  See Also:
  +
  Constant Field Values
  +
  +
+

@@ -206,27 +272,21 @@ extends java.lang.Object -

processFile

public static void processFile(java.io.File inputFile,
-                               java.io.File outputDir)
-                        throws java.lang.Exception

public static java.lang.String processFile(java.io.File input,
+                                           java.io.File outputDir)
+                                    throws java.lang.Exception

Reclaims all screenshots and maps found in a file or folder.

Returns:: what was reclaimed, for example "Reclaimed 2 pictures and 1 map"
Throws:: java.lang.Exception; java.lang.Exception - if nothing could be reclaimed

- -

getFileTypeDescription

public static java.lang.String getFileTypeDescription(java.io.File inputFile)

Prev Class
Next Class
Next Class

Frames

Detail:
Field |
Field |
Constr |
Method

Cortana (Halo 3 Reclaimer)

+ + +

Skip navigation links

+ + + +

Halo 3 Reclaimer

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

java.lang.Object
+
- Cortana
+

+
+
+
```
public class Cortana
+extends java.lang.Object
```
+
Reader for USB sticks set up by the Xbox 360. + +
The console doesn't store saves on the stick as normal files. It creates + a hidden folder Xbox360 with big files Data0000, + Data0001, Data0002... Joined together they are one + virtual drive in the Xbox's own FATX file system. Cortana opens that drive, + walks to the Halo 3 folders (Content\<profile>\4D5307E6\...) + and copies their files into a temporary folder, where the Arbiter sorts them + out like any other files. Nothing on the stick is ever changed.
+ +
Only Cortana can understand these needlesly complicated Michaelsoft file + systems.
+ +
Partition offsets and the FATX layout follow Free60 and Velocity + (hetelek/Velocity, GPLv3).
+
+
Version:
+
1.0
+
+

+ +

+ + +
Constructor Summary
+ + + + + + + + +
Constructors
Constructor and Description
Cortana()
+

Constructors
Constructor and Description
`Cortana()`

+ +

+ + +

Method Summary

+ + + + + + + + + + + + + + + + + + +

All Methods Static Methods Concrete Methods
Modifier and Type	Method and Description
`static java.io.File`	`copyHalo3Files(java.io.File storage)` + Copies every Halo 3 file from the stick into a new temporary folder and + returns that folder. +
`static void`	`deleteCopies(java.io.File folder)` + Deletes the temporary folder made by copyHalo3Files(). +
`static java.io.File`	`findStorage(java.io.File input)` + Returns the Xbox360 folder when the input is a USB stick set up by the + console, its Xbox360 folder or one of its Data files. +

+ + +
Methods inherited from class java.lang.Object
+clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

+ +
- + + +
  Constructor Detail
  + + + +
  - +
    Cortana
    +
```
public Cortana()
```
    +
  +
+ +
- + + +
  Method Detail
  + + + +
  - +
    findStorage
    +
```
public static java.io.File findStorage(java.io.File input)
```
    +
    Returns the Xbox360 folder when the input is a USB stick set up by the + console, its Xbox360 folder or one of its Data files. Otherwise null.
    +
  + + + +
  - +
    copyHalo3Files
    +
```
public static java.io.File copyHalo3Files(java.io.File storage)
+                                   throws java.lang.Exception
```
    +
    Copies every Halo 3 file from the stick into a new temporary folder and + returns that folder. Remove it with deleteCopies() when done.
    +
    +
    Throws:
    +
    java.lang.Exception - if the stick can't be read
    +
    +
  + + + +
  - +
    deleteCopies
    +
```
public static void deleteCopies(java.io.File folder)
```
    +
    Deletes the temporary folder made by copyHalo3Files().
    +
  +
+

+ + +

Skip navigation links

+ + + +

File extractor from Xbox 360 version of Halo 3

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

ErrorHandler (Halo 3 Reclaimer)

@@ -430,7 +430,7 @@ extends java.lang.Object

GraphicalUI (Halo 3 Reclaimer)

Version:: 1.0; 1.1
See Also:: Serialized Form

Main (Halo 3 Reclaimer)

MapReclaimer (Halo 3 Reclaimer)

public class MapReclaimer
 extends java.lang.Object

Core extraction engine for Forge map variants. - -

This is harder than it looks. WIP.

Halo 3 saves every Forge map as its own Xbox 360 STFS package + (a CON file with a long hex name). The package holds one file called + sandbox.map. That file is the Forge map variant itself: + the object placements on top of one of the original maps, stored in + Bungie's BLF format (chunks _blf, chdr, + mvar and _eof).

Unlike screenshots, the map has no magic bytes worth scanning for, + so this class reads the STFS file system properly: it walks the file + table, follows the block chain of sandbox.map and checks + each block against the package hash tables.

Supported input formats:

STFS container files (CON, LIVE, PIRS) holding a Halo 3 Forge map
already extracted sandbox.map files (BLF)

The saved file is not converted in any way. It is byte for byte the + sandbox.map the console keeps inside the package.

Version:: "1.0"; 1.0

@@ -156,7 +175,10 @@ extends java.lang.Object void

extractMap(java.io.File inputFile,
-          java.io.File outputDir)

+ java.io.File outputDir) +

Extracts the Forge map from an Xbox 360 file and saves it as + <input name>-reclaimed.map in the output directory.

public void extractMap(java.io.File inputFile,
                        java.io.File outputDir)
                 throws java.lang.Exception

Extracts the Forge map from an Xbox 360 file and saves it as + <input name>-reclaimed.map in the output directory.

Throws:: java.lang.Exception; java.lang.Exception - if the file cannot be read, is not a Halo 3 Forge map, + or is damaged

diff --git a/doc/ScreenshotReclaimer.html b/doc/ScreenshotReclaimer.html index 4dfb1db..f5d6983 100644 --- a/doc/ScreenshotReclaimer.html +++ b/doc/ScreenshotReclaimer.html @@ -2,9 +2,9 @@ - + ScreenshotReclaimer (Halo 3 Reclaimer) - + diff --git a/doc/TerminalUI.html b/doc/TerminalUI.html index 76098a3..ebfd352 100644 --- a/doc/TerminalUI.html +++ b/doc/TerminalUI.html @@ -2,9 +2,9 @@ - + TerminalUI (Halo 3 Reclaimer) - + diff --git a/doc/allclasses-frame.html b/doc/allclasses-frame.html index 47b87ab..9b46ad1 100644 --- a/doc/allclasses-frame.html +++ b/doc/allclasses-frame.html @@ -2,9 +2,9 @@ - + All Classes (Halo 3 Reclaimer) - + @@ -13,6 +13,7 @@

Arbiter
Cortana
ErrorHandler
GraphicalUI
Main

All Classes (Halo 3 Reclaimer)

Arbiter
Cortana
ErrorHandler
GraphicalUI
Main

Uses of Class Arbiter (Halo 3 Reclaimer)

Uses of Class Cortana (Halo 3 Reclaimer)

+ + +

Skip navigation links

+ + + +

Halo 3 Reclaimer

Frames
No Frames

All Classes

+ +

+ + +

No usage of Cortana

+ + +

Skip navigation links

+ + + +

File extractor from Xbox 360 version of Halo 3

Frames
No Frames

All Classes

+ +

+ + +

Uses of Class ErrorHandler (Halo 3 Reclaimer)

Uses of Class GraphicalUI (Halo 3 Reclaimer)

Uses of Class Main (Halo 3 Reclaimer)

Uses of Class MapReclaimer (Halo 3 Reclaimer)

Uses of Class ScreenshotReclaimer (Halo 3 Reclaimer)

Uses of Class TerminalUI (Halo 3 Reclaimer)

Constant Field Values (Halo 3 Reclaimer)

+ + + + + + + + + + + + + + + + + + +
Arbiter
Modifier and Type Constant Field Value
+ +public static final java.lang.String MAPS_DIR "maps-reclaimed"
+ +public static final java.lang.String SCREENSHOTS_DIR "screenshots-reclaimed"
+

Arbiter
Modifier and Type	Constant Field	Value
+ +`public static final java.lang.String`	`MAPS_DIR`	`"maps-reclaimed"`
+ +`public static final java.lang.String`	`SCREENSHOTS_DIR`	`"screenshots-reclaimed"`

+ diff --git a/doc/deprecated-list.html b/doc/deprecated-list.html index 9110a93..dd1a2bd 100644 --- a/doc/deprecated-list.html +++ b/doc/deprecated-list.html @@ -2,9 +2,9 @@ - +Deprecated List (Halo 3 Reclaimer) - + diff --git a/doc/help-doc.html b/doc/help-doc.html index b6b61fa..1bde3bc 100644 --- a/doc/help-doc.html +++ b/doc/help-doc.html @@ -2,9 +2,9 @@ - +API Help (Halo 3 Reclaimer) - + diff --git a/doc/index-all.html b/doc/index-all.html index 44be758..447e6e5 100644 --- a/doc/index-all.html +++ b/doc/index-all.html @@ -2,9 +2,9 @@ - +Index (Halo 3 Reclaimer) - + @@ -70,7 +70,7 @@ -

A B C E G M N P S T U +

A B C D E F G M N P S T U

A

@@ -99,6 +99,27 @@

CANT_PROCESS - Static variable in class ErrorHandler
copyHalo3Files(File) - Static method in class Cortana: +
Copies every Halo 3 file from the stick into a new temporary folder and + returns that folder.
+
Cortana - Class in <Unnamed>: +
Reader for USB sticks set up by the Xbox 360.
+
Cortana() - Constructor for class Cortana

+ + + +

D

deleteCopies(File) - Static method in class Cortana: +
Deletes the temporary folder made by copyHalo3Files().
+

@@ -120,19 +141,31 @@

TUI only: print error AND QUIT

extractMap(File, File) - Method in class MapReclaimer

Extracts the Forge map from an Xbox 360 file and saves it as + <input name>-reclaimed.map in the output directory.

extractScreenshot(File, File) - Method in class ScreenshotReclaimer

Extracts a screenshot from an Xbox 360 file and saves it as a JPEG.

+ + + +

F

findStorage(File) - Static method in class Cortana: +
Returns the Xbox360 folder when the input is a USB stick set up by the + console, its Xbox360 folder or one of its Data files.
+

G

getFileTypeDescription(File) - Static method in class Arbiter
GraphicalUI - Class in <Unnamed>: Graphical User Interface for Halo 3 Reclaimer.
@@ -169,6 +202,8 @@
MapReclaimer() - Constructor for class MapReclaimer
MAPS_DIR - Static variable in class Arbiter

@@ -186,7 +221,9 @@

P

processFile(File, File) - Static method in class Arbiter: +
Reclaims all screenshots and maps found in a file or folder.
+
processFile(String[]) - Static method in class TerminalUI: processes file extraction from command line.
@@ -203,6 +240,8 @@
ScreenshotReclaimer() - Constructor for class ScreenshotReclaimer
SCREENSHOTS_DIR - Static variable in class Arbiter
showHelp() - Static method in class TerminalUI
showTooManyArguments() - Static method in class TerminalUI
UNKNOWN_FILE - Static variable in class ErrorHandler

-A B C E G M N P S T U

+A B C D E F G M N P S T U

diff --git a/doc/index.html b/doc/index.html index 61486f1..37a8f0c 100644 --- a/doc/index.html +++ b/doc/index.html @@ -2,7 +2,7 @@ - + Halo 3 Reclaimer diff --git a/doc/overview-tree.html b/doc/overview-tree.html index 5ae38f8..e0bc245 100644 --- a/doc/overview-tree.html +++ b/doc/overview-tree.html @@ -2,9 +2,9 @@ - + Class Hierarchy (Halo 3 Reclaimer) - + @@ -100,6 +100,7 @@ +

diff --git a/doc/package-frame.html b/doc/package-frame.html index 43d5aa9..962c54b 100644 --- a/doc/package-frame.html +++ b/doc/package-frame.html @@ -2,9 +2,9 @@ - + <Unnamed> (Halo 3 Reclaimer) - + @@ -14,6 +14,7 @@

Classes

Arbiter
Cortana
ErrorHandler
GraphicalUI
Main

(Halo 3 Reclaimer)

+ + + + - + - + - + - + - +

ErrorHandler
Modifier and Type
Cortana	+ Reader for USB sticks set up by the Xbox 360. +
ErrorHandler	Simple error handling class.
GraphicalUI	Graphical User Interface for Halo 3 Reclaimer.
Main	Main entry point for the program.
MapReclaimer	Core extraction engine for Forge map variants.
ScreenshotReclaimer	Core extraction engine for screenshots.
TerminalUI	Terminal User Interface for Halo 3 Reclaimer. diff --git a/doc/package-tree.html b/doc/package-tree.html index 47b96cb..e1b8c80 100644 --- a/doc/package-tree.html +++ b/doc/package-tree.html @@ -2,9 +2,9 @@ - + Class Hierarchy (Halo 3 Reclaimer) - + @@ -100,6 +100,7 @@ + Cortana ErrorHandler Main MapReclaimer diff --git a/doc/package-use.html b/doc/package-use.html index 9ac8125..72967b0 100644 --- a/doc/package-use.html +++ b/doc/package-use.html @@ -2,9 +2,9 @@ - + Uses of Package (Halo 3 Reclaimer) - + diff --git a/doc/serialized-form.html b/doc/serialized-form.html index e00571d..98c2f06 100644 --- a/doc/serialized-form.html +++ b/doc/serialized-form.html @@ -2,9 +2,9 @@ - + Serialized Form (Halo 3 Reclaimer) - + diff --git a/doc/user-manual-editable.odt b/doc/user-manual-editable.odt index 1f2e608..9bcb026 100644 Binary files a/doc/user-manual-editable.odt and b/doc/user-manual-editable.odt differ diff --git a/doc/user-manual.pdf b/doc/user-manual.pdf index 8f832a6..4a98f7c 100644 Binary files a/doc/user-manual.pdf and b/doc/user-manual.pdf differ diff --git a/halo3-reclaimer.jar b/halo3-reclaimer.jar index dae1f0d..990df86 100644 Binary files a/halo3-reclaimer.jar and b/halo3-reclaimer.jar differ diff --git a/src/Arbiter.java b/src/Arbiter.java index 6fa6706..b260dc4 100644 --- a/src/Arbiter.java +++ b/src/Arbiter.java @@ -1,60 +1,303 @@ /* * Halo 3 Reclaimer [Arbiter.java] -* player data extractor from Xbox 360 version of Halo 3 -* copyright 2025, David Polakovic, licensed under GPLv3 +* player data extractor from Xbox 360 version of Halo 3 +* copyright 2025, 2026, David Polakovic, licensed under GPLv3 * contact: email@dpolakovic.space / +import java.io.DataInputStream; import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; /* * Arbiter class that determines file type and routes to appropriate extractor. -* -* This class analyzes the input file to determine whether it's a screenshot -* or map file, then calls the appropriate reclaimer. The Xbox 360 saves file -* type directly in file name. So it just checks for names... -* -* @version 1.0 +* +* The input is a single file, a whole folder, or a USB stick set up by the +* Xbox 360. A stick (or its Xbox360 folder, or one of its Data files) goes to +* Cortana first, which copies the Halo 3 files out of the Xbox file system; +* other files on the stick are ignored then. +* +* Halo 3 keeps every screenshot and every Forge map in its own package, +* so the Arbiter looks inside each file: a package holding +* `screen.shot` is a screenshot, one holding `sandbox.map` +* is a map. Screenshots are checked first, then maps. Anything else is skipped. +* Only when the content says nothing (DDS, TGA, raw textures) the file name +* decides, like in version 1.0. +* +* Results go to two folders in the output directory: +* `screenshots-reclaimed` and `maps-reclaimed`. +* +* @version 1.1 / public class Arbiter { - - public static void processFile(File inputFile, File outputDir) throws Exception { - String fileName = inputFile.getName().toLowerCase(); - - if (isScreenshotFile(fileName)) { - // route to screenshot extractor - ScreenshotReclaimer screenshotReclaimer = new ScreenshotReclaimer(); - screenshotReclaimer.extractScreenshot(inputFile, outputDir); - } else if (isMapFile(fileName)) { - // route to map extractor - MapReclaimer mapReclaimer = new MapReclaimer(); - mapReclaimer.extractMap(inputFile, outputDir); - } else { - // unrecognized file type - throw ErrorHandler.makeException(ErrorHandler.UNKNOWN_FILE); - } - } - + + // folders created inside the output directory + public static final String SCREENSHOTS_DIR = "screenshots-reclaimed"; + public static final String MAPS_DIR = "maps-reclaimed"; + + // the file inside a package that holds a screenshot + private static final String SCREENSHOT_FILE = "screen.shot"; + + // packages are small, bigger files are not worth reading + private static final long MAX_FILE_SIZE = 16L 1024 * 1024; + + // how deep to look into sub folders + private static final int MAX_FOLDER_DEPTH = 8; + + private enum Kind { SCREENSHOT, MAP, OTHER } + + /** + * Reclaims all screenshots and maps found in a file or folder. + * + * @return what was reclaimed, for example "Reclaimed 2 pictures and 1 map" + * @throws Exception if nothing could be reclaimed + / + public static String processFile(File input, File outputDir) throws Exception { + // a USB stick set up by the console: Cortana gets the Halo 3 files out of it first + File storage = Cortana.findStorage(input); + if (storage != null) { + File copies = Cortana.copyHalo3Files(storage); + try { + return reclaim(listFiles(copies), outputDir, true, copies); + } finally { + Cortana.deleteCopies(copies); + } + } + + if (input.isDirectory()) { + return reclaim(listFiles(input), outputDir, true, input); + } + List files = new ArrayList(); + files.add(input); + return reclaim(files, outputDir, false, null); + } + + /* + * Sorts the files out and runs the right reclaimer on each, screenshots first. + * folder: there are many files to go through, root: where console paths start. + / + private static String reclaim(List files, File outputDir, boolean folder, File root) throws Exception { + // sort everything out first, screenshots go first + List screenshots = new ArrayList(); + List maps = new ArrayList(); + String skipReason = null; + for (File file : files) { + StringBuilder reason = new StringBuilder(); + StringBuilder detail = new StringBuilder(); + Kind kind = detect(file, reason, detail); + if (kind == Kind.SCREENSHOT) { + screenshots.add(file); + } else if (kind == Kind.MAP) { + maps.add(file); + } else { + // short reason for the status bar, the detail only goes to the console + skipReason = reason.toString(); + if (folder) { + System.out.println("Skipping " + shown(root, file) + " (" + + (detail.length() > 0 ? detail : skipReason) + ")"); + } else if (detail.length() > 0) { + System.out.println("Not reclaimed: " + detail); + } + } + } + + if (screenshots.isEmpty() && maps.isEmpty()) { + if (folder) { + throw ErrorHandler.makeException(ErrorHandler.UNKNOWN_FILE + ": no Halo 3 files found"); + } + throw ErrorHandler.makeException(skipReason); + } + if (folder) { + System.out.println("Found " + count(screenshots.size(), "screenshot") + + " and " + count(maps.size(), "map")); + } + + int failed = 0; + Exception lastError = null; + + int pictures = 0; + if (!screenshots.isEmpty()) { + File screenshotsDir = outputFolder(outputDir, SCREENSHOTS_DIR); + for (File file : screenshots) { + System.out.println("Detected as screenshot: " + shown(root, file)); + try { + new ScreenshotReclaimer().extractScreenshot(file, screenshotsDir); + pictures++; + } catch (Exception e) { + failed++; + lastError = e; + System.out.println("Could not reclaim " + shown(root, file) + ": " + e.getMessage()); + } + } + } + + int reclaimedMaps = 0; + if (!maps.isEmpty()) { + File mapsDir = outputFolder(outputDir, MAPS_DIR); + for (File file : maps) { + System.out.println("Detected as Forge map: " + shown(root, file)); + try { + new MapReclaimer().extractMap(file, mapsDir); + reclaimedMaps++; + } catch (Exception e) { + failed++; + lastError = e; + System.out.println("Could not reclaim " + shown(root, file) + ": " + e.getMessage()); + } + } + } + + if (pictures + reclaimedMaps == 0) { + if (failed == 1) { + throw lastError; + } + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": all " + failed + " files failed"); + } + + List reclaimed = new ArrayList(); + if (pictures > 0) reclaimed.add(count(pictures, "picture")); + if (reclaimedMaps > 0) reclaimed.add(count(reclaimedMaps, "map")); + String result = "Reclaimed " + String.join(" and ", reclaimed); + if (failed > 0) { + result += " (" + failed + " failed)"; + } + return result; + } + + /* + * Tells what a file is by looking inside it. Screenshots are checked first. + * When the file is neither, a short error goes into reason and, when + * there is more to say, an explanation for the console into detail. + / + private static Kind detect(File file, StringBuilder reason, StringBuilder detail) { + if (file.length() > MAX_FILE_SIZE) { + reason.append(ErrorHandler.CANT_PROCESS + ": file too big"); + detail.append(file.length() + " bytes, too big for a screenshot or map"); + return Kind.OTHER; + } + + byte[] data; + try { + data = readFile(file); + } catch (IOException e) { + reason.append(ErrorHandler.BAD_INPUT + ": " + e.getMessage()); + return Kind.OTHER; + } + + if (MapReclaimer.isSTFSFile(data)) { + try { + List names = MapReclaimer.listPackageFiles(data); + if (containsName(names, SCREENSHOT_FILE)) return Kind.SCREENSHOT; + if (containsName(names, MapReclaimer.MAP_VARIANT_FILE)) return Kind.MAP; + // game types, films... + reason.append(ErrorHandler.CANT_PROCESS + ": not a screenshot or map"); + detail.append(names.isEmpty() ? "the package is empty" + : "the package holds " + String.join(", ", names)); + return Kind.OTHER; + } catch (Exception e) { + // another game or a damaged package, the name may still tell + reason.append(e.getMessage() != null ? e.getMessage() : ErrorHandler.CANT_PROCESS); + } + } else if (MapReclaimer.isBLFFile(data)) { + // screen.shot or sandbox.map already taken out of its package + int fileType = MapReclaimer.readBlfFileType(data); + if (fileType == MapReclaimer.FILE_TYPE_SCREENSHOT) return Kind.SCREENSHOT; + if (fileType == MapReclaimer.FILE_TYPE_MAP_VARIANT) return Kind.MAP; + if (fileType >= 0) { + reason.append(ErrorHandler.CANT_PROCESS + ": not a screenshot or map"); + detail.append("Halo file of type " + fileType + " (game type, film...)"); + return Kind.OTHER; + } + } + + // the content says nothing (DDS, TGA, raw textures), let the name decide + String fileName = file.getName().toLowerCase(); + if (isScreenshotFile(fileName)) return Kind.SCREENSHOT; + if (isMapFile(fileName)) return Kind.MAP; + + if (reason.length() == 0) { + reason.append(ErrorHandler.UNKNOWN_FILE); + } + return Kind.OTHER; + } private static boolean isScreenshotFile(String fileName) { return fileName.contains("screenshot") \|\| fileName.contains("shot"); } - private static boolean isMapFile(String fileName) { return fileName.contains("map") \|\| fileName.contains("variant") \|\| fileName.contains("forge"); } - - public static String getFileTypeDescription(File inputFile) { - String fileName = inputFile.getName().toLowerCase(); - - if (isScreenshotFile(fileName)) { - return "Screenshot file"; - } else if (isMapFile(fileName)) { - return "Map file"; - } else { - return "Unrecognized file"; + + private static boolean containsName(List names, String wanted) { + for (String name : names) { + if (name.equalsIgnoreCase(wanted)) return true; + } + return false; + } + + private static List listFiles(File folder) { + List files = new ArrayList(); + collectFiles(folder, files, 0); + return files; + } + + // path for the console, starting at the picked folder + private static String shown(File root, File file) { + if (root == null) return file.getPath(); + try { + return root.toPath().relativize(file.toPath()).toString(); + } catch (IllegalArgumentException e) { + return file.getPath(); } } -} \ No newline at end of file + + /* + * Adds all files of a folder and its sub folders, skipping our own output folders. + / + private static void collectFiles(File folder, List files, int depth) { + if (depth > MAX_FOLDER_DEPTH) return; + File[] children = folder.listFiles(); + if (children == null) return; + Arrays.sort(children); + for (File child : children) { + if (child.isDirectory()) { + String name = child.getName(); + if (!name.equals(SCREENSHOTS_DIR) && !name.equals(MAPS_DIR)) { + collectFiles(child, files, depth + 1); + } + } else if (child.isFile()) { + files.add(child); + } + } + } + + /* + * Returns the output sub folder, creates it when needed. + / + private static File outputFolder(File outputDir, String name) throws Exception { + File folder = new File(outputDir, name); + if (!folder.isDirectory() && !folder.mkdirs()) { + throw ErrorHandler.makeException(ErrorHandler.BAD_OUTPUT + ": " + folder.getPath()); + } + return folder; + } + + private static byte[] readFile(File file) throws IOException { + byte[] data = new byte[(int) file.length()]; + try (DataInputStream in = new DataInputStream(new FileInputStream(file))) { + in.readFully(data); + } + return data; + } + + // "1 map", "2 maps" + private static String count(int number, String word) { + return number + " " + word + (number == 1 ? "" : "s"); + } +} diff --git a/src/Cortana.java b/src/Cortana.java new file mode 100644 index 0000000..ac32a1e --- /dev/null +++ b/src/Cortana.java @@ -0,0 +1,454 @@ +/ +* Halo 3 Reclaimer [Cortana.java] +* player data extractor from Xbox 360 version of Halo 3 +* copyright 2026, David Polakovic, licensed under GPLv3 +* contact: email@dpolakovic.space +/ + +import java.io.Closeable; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.RandomAccessFile; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.List; + +/* +* Reader for USB sticks set up by the Xbox 360. +* +* The console doesn't store saves on the stick as normal files. It creates +* a hidden folder `Xbox360` with big files `Data0000`, +* `Data0001`, `Data0002`... Joined together they are one +* virtual drive in the Xbox's own FATX file system. Cortana opens that drive, +* walks to the Halo 3 folders (`Content\<profile>\4D5307E6\...`) +* and copies their files into a temporary folder, where the Arbiter sorts them +* out like any other files. Nothing on the stick is ever changed. +* +* Only Cortana can understand these needlesly complicated Michaelsoft file +* systems. +* +* Partition offsets and the FATX layout follow Free60 and Velocity +* (hetelek/Velocity, GPLv3). +* +* @version 1.0 +/ + +public class Cortana { + + // the hidden folder the console creates on the stick + private static final String STORAGE_FOLDER = "Xbox360"; + + // Halo 3 keeps its files in folders named after its title ID + private static final String HALO3_FOLDER = "4D5307E6"; + + // user content starts 512 MB into the joined Data files + private static final long CONTENT_PARTITION = 0x20000000L; + + // FATX format constants + private static final int FATX_MAGIC = 0x58544146; // "XTAF" + private static final int FATX_HEADER_SIZE = 0x1000; // the allocation table follows it + private static final int ENTRY_SIZE = 0x40; + private static final int ENTRY_NAME_LENGTH = 0x2A; + private static final int ENTRY_DELETED = 0xE5; + private static final int ATTRIBUTE_DIRECTORY = 0x10; + + // screenshots and maps are small, add-ons and updates stay on the stick + private static final long MAX_FILE_SIZE = 16L 1024 * 1024; + + // how deep to look for the Halo 3 folders, and how long a folder may get + private static final int MAX_DEPTH = 6; + private static final int MAX_FOLDER_CLUSTERS = 1024; + + /** + * Returns the Xbox360 folder when the input is a USB stick set up by the + * console, its Xbox360 folder or one of its Data files. Otherwise null. + / + public static File findStorage(File input) { + if (input == null) return null; + + if (input.isFile()) { + // one of the Data files was picked + File parent = input.getParentFile(); + if (input.getName().matches("(?i)data\\d{4}") && parent != null && hasDataFiles(parent)) { + return parent; + } + return null; + } + + if (input.isDirectory()) { + if (hasDataFiles(input)) return input; + File storage = findChild(input, STORAGE_FOLDER); + if (storage != null && storage.isDirectory() && hasDataFiles(storage)) { + return storage; + } + } + return null; + } + + /* + * Copies every Halo 3 file from the stick into a new temporary folder and + * returns that folder. Remove it with deleteCopies() when done. + * + * @throws Exception if the stick can't be read + / + public static File copyHalo3Files(File storage) throws Exception { + List dataFiles = listDataFiles(storage); + System.out.println("Reading Xbox 360 USB storage: " + storage.getPath() + + " (" + dataFiles.size() + " Data files)"); + + File copies; + try { + copies = Files.createTempDirectory("halo3-reclaimer-").toFile(); + } catch (IOException e) { + throw ErrorHandler.makeException(ErrorHandler.BAD_OUTPUT + ": no temporary folder"); + } + + try (Drive drive = new Drive(dataFiles)) { + if (drive.length <= CONTENT_PARTITION + FATX_HEADER_SIZE) { + throw unreadable("the Data files hold only " + drive.length + " bytes"); + } + Volume content = new Volume(drive, CONTENT_PARTITION, drive.length - CONTENT_PARTITION, true); + int copied = copyFolder(content, content.root(), copies, "", false, 0); + System.out.println("Copied " + copied + " Halo 3 file(s) off the stick"); + } catch (RuntimeException e) { + // a bug or a layout nobody has seen yet + deleteCopies(copies); + e.printStackTrace(); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": USB stick not readable"); + } catch (IOException e) { + deleteCopies(copies); + throw ErrorHandler.makeException(ErrorHandler.BAD_INPUT + ": " + e.getMessage()); + } catch (Exception e) { + deleteCopies(copies); + throw e; + } + return copies; + } + + /* + * Deletes the temporary folder made by copyHalo3Files(). + / + public static void deleteCopies(File folder) { + if (folder == null) return; + File[] children = folder.listFiles(); + if (children != null) { + for (File child : children) { + if (child.isDirectory()) { + deleteCopies(child); + } else { + child.delete(); + } + } + } + folder.delete(); + } + + /* + * Walks the folders. Inside a 4D5307E6 folder every file is copied out. + / + private static int copyFolder(Volume volume, Entry folder, File target, String path, + boolean inHalo3, int depth) throws Exception { + if (depth > MAX_DEPTH) return 0; + + int copied = 0; + for (Entry entry : volume.list(folder)) { + String entryPath = path.isEmpty() ? entry.name : path + "\\" + entry.name; + File entryTarget = new File(target, safeName(entry.name)); + + if (entry.isDirectory()) { + boolean halo3 = inHalo3 \|\| entry.name.equalsIgnoreCase(HALO3_FOLDER); + if (halo3 && !inHalo3) { + System.out.println("Found Halo 3 folder: " + entryPath); + } + copied += copyFolder(volume, entry, entryTarget, entryPath, halo3, depth + 1); + } else if (inHalo3) { + if (entry.size > MAX_FILE_SIZE) { + System.out.println("Leaving " + entryPath + " on the stick (" + entry.size + + " bytes, too big for a screenshot or map)"); + continue; + } + writeFile(entryTarget, volume.read(entry)); + copied++; + } + } + return copied; + } + + /* + * Data0000, Data0001... in order, until one is missing. + / + private static List listDataFiles(File storage) { + List files = new ArrayList(); + for (int i = 0; i < 10000; i++) { + File data = findChild(storage, String.format("Data%04d", i)); + if (data == null \|\| !data.isFile()) break; + files.add(data); + } + return files; + } + + private static boolean hasDataFiles(File folder) { + File first = findChild(folder, "Data0000"); + return first != null && first.isFile(); + } + + /* + * Finds a file or folder by name, ignoring upper and lower case. + / + private static File findChild(File folder, String name) { + File exact = new File(folder, name); + if (exact.exists()) return exact; + String[] names = folder.list(); + if (names == null) return null; + for (String candidate : names) { + if (candidate.equalsIgnoreCase(name)) return new File(folder, candidate); + } + return null; + } + + private static void writeFile(File file, byte[] data) throws Exception { + File parent = file.getParentFile(); + if (parent != null && !parent.isDirectory() && !parent.mkdirs()) { + throw ErrorHandler.makeException(ErrorHandler.BAD_OUTPUT + ": no temporary folder"); + } + try (FileOutputStream out = new FileOutputStream(file)) { + out.write(data); + } catch (IOException e) { + throw ErrorHandler.makeException(ErrorHandler.BAD_OUTPUT + ": " + e.getMessage()); + } + } + + // Xbox names are plain, but never trust them with the PC's file system + private static String safeName(String name) { + String safe = name.replaceAll("[\\\\/:?\"<>\|\\p{Cntrl}]", "_").trim(); + return (safe.isEmpty() \|\| safe.equals(".") \|\| safe.equals("..")) ? "_" : safe; + } + + // short message for the status bar, the reason goes to the console + private static Exception damaged(String reason) { + System.out.println("Damaged USB stick: " + reason); + return ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": damaged USB stick"); + } + + private static Exception unreadable(String reason) { + System.out.println("USB stick not readable: " + reason); + return ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": USB stick not readable"); + } + + private static int readInt32(byte[] data, int offset) { + return ((data[offset] & 0xFF) << 24) \| ((data[offset + 1] & 0xFF) << 16) + \| ((data[offset + 2] & 0xFF) << 8) \| (data[offset + 3] & 0xFF); + } + + /** + * One file or folder on the Xbox drive. + / + private static final class Entry { + String name; + int attributes; + long firstCluster; + long size; + + boolean isDirectory() { + return (attributes & ATTRIBUTE_DIRECTORY) != 0; + } + } + + /* + * The Data files joined into one long drive. Only the needed parts are read, + * the files can be gigabytes. + / + private static final class Drive implements Closeable { + private final RandomAccessFile[] files; + private final long[] starts; + private final long[] lengths; + final long length; + + Drive(List dataFiles) throws IOException { + files = new RandomAccessFile[dataFiles.size()]; + starts = new long[files.length]; + lengths = new long[files.length]; + long total = 0; + try { + for (int i = 0; i < files.length; i++) { + files[i] = new RandomAccessFile(dataFiles.get(i), "r"); + starts[i] = total; + lengths[i] = files[i].length(); + total += lengths[i]; + } + } catch (IOException e) { + close(); + throw e; + } + length = total; + } + + /* + * Reads count bytes at a position of the joined drive. + / + byte[] read(long position, int count) throws Exception { + if (position < 0 \|\| count < 0 \|\| position + count > length) { + throw damaged("something points past the end of the Data files"); + } + byte[] out = new byte[count]; + int done = 0; + int index = 0; + while (done < count) { + long pos = position + done; + while (pos >= starts[index] + lengths[index]) { + index++; + } + long inFile = pos - starts[index]; + int chunk = (int) Math.min(count - done, lengths[index] - inFile); + files[index].seek(inFile); + files[index].readFully(out, done, chunk); + done += chunk; + } + return out; + } + + @Override + public void close() { + for (RandomAccessFile file : files) { + if (file != null) { + try { + file.close(); + } catch (IOException e) { + // nothing left to do + } + } + } + } + } + + /* + * One FATX partition: a 4 KB header, the allocation table, then the clusters. + / + private static final class Volume { + private final Drive drive; + private final long rootCluster; + private final int clusterSize; + private final int fatEntrySize; // 2 or 4 bytes + private final long fatStart; + private final long clusterStart; + private final long clusterCount; + + Volume(Drive drive, long start, long size, boolean usbContent) throws Exception { + this.drive = drive; + + // header: magic, partition ID, sectors per cluster, root folder cluster + byte[] header = drive.read(start, 16); + if (readInt32(header, 0) != FATX_MAGIC) { + throw unreadable("no Xbox file system where the content should start"); + } + int sectorsPerCluster = readInt32(header, 8); + rootCluster = readInt32(header, 12) & 0xFFFFFFFFL; + switch (sectorsPerCluster) { + case 0x2: case 0x4: case 0x8: case 0x10: case 0x20: case 0x40: case 0x80: + break; + default: + throw damaged("unknown cluster size"); + } + clusterSize = sectorsPerCluster << 9; + int shift = Integer.numberOfTrailingZeros(clusterSize); + + // size of the allocation table, worked out the same way as the console does; + // the content partition of a USB stick always uses 4 byte entries + long clusters = (size >> shift) + 1; + int entryShift = (usbContent \|\| clusters >= 0xFFF0) ? 2 : 1; + long fatSize = ((clusters << entryShift) + 0xFFF) & ~0xFFFL & 0xFFFFFFFFL; + long rest = size - FATX_HEADER_SIZE; + if (rest < fatSize) { + throw damaged("partition too small for its allocation table"); + } + + fatEntrySize = 1 << entryShift; + fatStart = start + FATX_HEADER_SIZE; + clusterStart = fatStart + fatSize; + clusterCount = (rest - fatSize) >> shift; + } + + Entry root() { + Entry root = new Entry(); + root.name = ""; + root.attributes = ATTRIBUTE_DIRECTORY; + root.firstCluster = rootCluster; + return root; + } + + /* + * Files and folders inside a folder, deleted ones left out. + / + List list(Entry folder) throws Exception { + List entries = new ArrayList(); + long cluster = folder.firstCluster; + int visited = 0; + while (isCluster(cluster)) { + if (++visited > MAX_FOLDER_CLUSTERS) { + throw damaged("a folder never ends"); + } + byte[] data = drive.read(clusterOffset(cluster), clusterSize); + for (int pos = 0; pos + ENTRY_SIZE <= data.length; pos += ENTRY_SIZE) { + // entry: name length, attributes, 42 byte name, first cluster, size, 3 dates + int nameLength = data[pos] & 0xFF; + if (nameLength == 0x00 \|\| nameLength == 0xFF) { + return entries; // end of the folder + } + if (nameLength == ENTRY_DELETED \|\| nameLength > ENTRY_NAME_LENGTH) { + continue; // deleted + } + Entry entry = new Entry(); + entry.attributes = data[pos + 1] & 0xFF; + entry.name = new String(data, pos + 2, nameLength, StandardCharsets.ISO_8859_1); + entry.firstCluster = readInt32(data, pos + 0x2C) & 0xFFFFFFFFL; + entry.size = readInt32(data, pos + 0x30) & 0xFFFFFFFFL; + entries.add(entry); + } + cluster = nextCluster(cluster); + } + return entries; + } + + /* + * The contents of a file, following its cluster chain. + / + byte[] read(Entry file) throws Exception { + byte[] out = new byte[(int) file.size]; + long cluster = file.firstCluster; + int done = 0; + while (done < out.length) { + if (!isCluster(cluster)) { + throw damaged("a file is cut off"); + } + int chunk = Math.min(clusterSize, out.length - done); + byte[] part = drive.read(clusterOffset(cluster), chunk); + System.arraycopy(part, 0, out, done, chunk); + done += chunk; + if (done < out.length) { + cluster = nextCluster(cluster); + } + } + return out; + } + + // clusters count from 1; free (0) and end markers fall outside this range + private boolean isCluster(long cluster) { + return cluster >= 1 && cluster <= clusterCount; + } + + private long clusterOffset(long cluster) { + return clusterStart + (cluster - 1) clusterSize; + } + + // the allocation table holds the next cluster of every cluster + private long nextCluster(long cluster) throws Exception { + byte[] entry = drive.read(fatStart + cluster * fatEntrySize, fatEntrySize); + if (fatEntrySize == 4) { + return readInt32(entry, 0) & 0xFFFFFFFFL; + } + return ((entry[0] & 0xFF) << 8) \| (entry[1] & 0xFF); + } + } +} diff --git a/src/GraphicalUI.java b/src/GraphicalUI.java index f51ba1b..08112f6 100644 --- a/src/GraphicalUI.java +++ b/src/GraphicalUI.java @@ -1,7 +1,7 @@ /* * Halo 3 Reclaimer [GraphicalUI.java] * player data extractor from Xbox 360 version of Halo 3 -* copyright 2025, David Polakovic, licensed under GPLv3 +* copyright 2025, 2026, David Polakovic, licensed under GPLv3 * contact: email@dpolakovic.space / @@ -17,7 +17,7 @@ import java.io.File; This class contains all the GUI components and * logic for the application's windowed interface * -* @version 1.0 +* @version 1.1 / public class GraphicalUI extends JFrame { @@ -104,7 +104,7 @@ public class GraphicalUI extends JFrame { // the action button JPanel buttonPanel = new JPanel(new FlowLayout(FlowLayout.CENTER)); buttonPanel.setBorder(BorderFactory.createEmptyBorder(0, 10, 0, 0)); - reclaimButton = new JButton("PELICAN!"); + reclaimButton = new JButton("reclaim"); reclaimButton.setPreferredSize(new Dimension(120, 40)); reclaimButton.setFont(new Font(Font.SANS_SERIF, Font.BOLD, 14)); buttonPanel.add(reclaimButton); @@ -161,9 +161,9 @@ public class GraphicalUI extends JFrame { / private void showVersionInfo() { String versionText = "Halo 3 Reclaimer\n\n" + - "Version: 1.0\n\n" + + "Version: 0.2\n\n" + "player data extractor from Xbox 360 version of Halo 3\n" + - "copyright 2025, David Polakovic, licensed under GPLv3\n"; + "copyright 2025, 2026, David Polakovic, licensed under GPLv3\n"; JOptionPane.showMessageDialog(this, versionText, "Version Information", JOptionPane.INFORMATION_MESSAGE); @@ -178,11 +178,11 @@ public class GraphicalUI extends JFrame { " to select your Xbox 360 Halo 3 file\n\n" + "2. Click 'Browse' next to 'Save destination'\n" + " to choose where to save the converted image\n\n" + - "3. Click 'PELICAN!' to start the extraction process\n\n" + - "The tool will convert your Xbox 360 file\n" + - "into a standard JPEG image or map file\n" + - "that you share on any device.\n\n" + - " map files are not implemented yet"; + "3. Click 'reclaim' to start the extraction process\n\n" + + "The tool will extract your user data from the\n" + + "Xbox 360 save. Currently supports:\n" + + "-screenshots\n" + + "-Forge maps\n\n"; JOptionPane.showMessageDialog(this, helpText, "Help", JOptionPane.INFORMATION_MESSAGE); @@ -226,8 +226,9 @@ public class GraphicalUI extends JFrame { @Override public void actionPerformed(ActionEvent e) { JFileChooser fileChooser = new JFileChooser(); - fileChooser.setFileSelectionMode(JFileChooser.FILES_ONLY); - fileChooser.setDialogTitle("Select Halo 3 raw file"); + fileChooser.setFileSelectionMode(JFileChooser.FILES_AND_DIRECTORIES); + fileChooser.setFileHidingEnabled(false); // the Xbox360 folder on a stick is hidden + fileChooser.setDialogTitle("Select Halo 3 file or folder"); // set file filter for Xbox file formats fileChooser.addChoosableFileFilter(new javax.swing.filechooser.FileFilter() { @@ -318,36 +319,28 @@ public class GraphicalUI extends JFrame { resetStatusColor(); setStatus("Processing file..."); - SwingWorker worker = new SwingWorker() { + SwingWorker worker = new SwingWorker() { @Override - protected Void doInBackground() throws Exception { + protected String doInBackground() throws Exception { try { - Arbiter.processFile(rawFile, saveDir); + return Arbiter.processFile(rawFile, saveDir); } catch (Exception ex) { SwingUtilities.invokeLater(() -> { setErrorStatus(ex.getMessage()); }); throw ex; } - return null; } @Override protected void done() { reclaimButton.setEnabled(true); - reclaimButton.setText("PELICAN!"); + reclaimButton.setText("reclaim"); try { - get(); // Check if any exception occurred - - String baseName = rawFile.getName(); - int dotIndex = baseName.lastIndexOf('.'); - if (dotIndex > 0) { - baseName = baseName.substring(0, dotIndex); - } - String outputFileName = baseName + "-reclaimed.jpg"; - - setSuccessStatus("Success! Saved as: " + outputFileName); + // e.g. "Reclaimed 2 pictures and 1 map", throws if any exception occurred + String result = get(); + setSuccessStatus("Success! " + result); } catch (Exception ex) { } } }; @@ -355,4 +348,4 @@ public class GraphicalUI extends JFrame { worker.execute(); } } -} \ No newline at end of file +} diff --git a/src/MapReclaimer.java b/src/MapReclaimer.java index c95c256..9a136c6 100644 --- a/src/MapReclaimer.java +++ b/src/MapReclaimer.java @@ -1,24 +1,648 @@ /* * Halo 3 Reclaimer [MapReclaimer.java] -* player data extractor from Xbox 360 version of Halo 3 -* copyright 2025, David Polakovic, licensed under GPLv3 +* player data extractor from Xbox 360 version of Halo 3 +* copyright 2025, 2026, David Polakovic, licensed under GPLv3 * contact: email@dpolakovic.space / +import java.io.DataInputStream; import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; /* * Core extraction engine for Forge map variants. -* -* This is harder than it looks. WIP. -* -* @version "1.0" +* +* Halo 3 saves every Forge map as its own Xbox 360 STFS package +* (a CON file with a long hex name). The package holds one file called +* `sandbox.map`. That file is the Forge map variant itself: +* the object placements on top of one of the original maps, stored in +* Bungie's BLF format (chunks `_blf`, `chdr`, +* `mvar` and `_eof`). +* +* Unlike screenshots, the map has no magic bytes worth scanning for, +* so this class reads the STFS file system properly: it walks the file +* table, follows the block chain of `sandbox.map` and checks +* each block against the package hash tables. +* +* Supported input formats: +* +* STFS container files (CON, LIVE, PIRS) holding a Halo 3 Forge map +* already extracted `sandbox.map` files (BLF) +* +* +* The saved file is not converted in any way. It is byte for byte the +* `sandbox.map` the console keeps inside the package. +* +* @version 1.0 / public class MapReclaimer { - + // every Halo 3 package carries this title ID in its header + private static final int HALO3_TITLE_ID = 0x4D5307E6; + + // a Forge map package is around 100 KB, anything this big is something else + private static final long MAX_INPUT_SIZE = 16L 1024 * 1024; + + // STFS format constants + private static final int STFS_MAGIC_CON = 0x434F4E20; // "CON " + private static final int STFS_MAGIC_LIVE = 0x4C495645; // "LIVE" + private static final int STFS_MAGIC_PIRS = 0x50495253; // "PIRS" + private static final int STFS_BLOCK_SIZE = 0x1000; + private static final int STFS_BLOCKS_PER_TABLE = 0xAA; // data blocks covered by one hash table + private static final int STFS_BLOCKS_PER_LEVEL1 = 0x70E4; // data blocks covered by one level 1 table + private static final int STFS_HASH_ENTRY_SIZE = 0x18; + private static final int STFS_FILE_ENTRY_SIZE = 0x40; + + // the file inside the package that holds the Forge map + static final String MAP_VARIANT_FILE = "sandbox.map"; + + // BLF format constants (the format of sandbox.map) + private static final int BLF_START = 0x5F626C66; // "_blf" + private static final int BLF_END = 0x5F656F66; // "_eof" + private static final int BLF_CONTENT_HEADER = 0x63686472; // "chdr" + private static final int BLF_MAP_VARIANT = 0x6D766172; // "mvar", Xbox 360 + private static final int BLF_MAP_VARIANT_PC = 0x6D617076; // "mapv", PC builds of the engine + private static final int BLF_CHUNK_HEADER_SIZE = 12; + + // item metadata (name, author...) sits 0x10 bytes into both chdr and mvar, + // these offsets are relative to its start + private static final int METADATA_OFFSET = 0x10; + private static final int METADATA_NAME = 0x08; // UTF-16, 16 characters + private static final int METADATA_DESCRIPTION = 0x28; // UTF-8, 128 bytes + private static final int METADATA_AUTHOR = 0xA8; // 16 bytes + private static final int METADATA_FILE_TYPE = 0xB8; + private static final int METADATA_MAP_ID = 0xE0; + + // saved game file types as stored in the metadata + static final int FILE_TYPE_MAP_VARIANT = 10; + static final int FILE_TYPE_SCREENSHOT = 13; + private static final String[] FILE_TYPE_NAMES = { + "campaign save", "Capture the Flag game type", "Slayer game type", + "Oddball game type", "King of the Hill game type", "Juggernaut game type", + "Territories game type", "Assault game type", "Infection game type", + "VIP game type", "Forge map", "saved film", "film clip", "screenshot" + }; + + /** + * Extracts the Forge map from an Xbox 360 file and saves it as + * <input name>-reclaimed.map in the output directory. + * + * @throws Exception if the file cannot be read, is not a Halo 3 Forge map, + * or is damaged + / public void extractMap(File inputFile, File outputDir) throws Exception { - throw ErrorHandler.makeException(ErrorHandler.NOT_READY); + byte[] fileData = readFile(inputFile); + + System.out.println("File size: " + fileData.length + " bytes"); + + byte[] mapData; + MapInfo info; + try { + if (isSTFSFile(fileData)) { + System.out.println("Detected STFS file format"); + mapData = extractFromSTFS(fileData); + } else if (isBLFFile(fileData)) { + System.out.println("Detected extracted map variant (BLF) file"); + mapData = fileData; + } else { + System.out.println("Not an Xbox 360 package or map variant"); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": not a Forge map"); + } + info = readMapInfo(mapData); + } catch (RuntimeException e) { + // a bug or a file layout nobody has seen yet, keep the GUI informed + e.printStackTrace(); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": damaged or unsupported file"); + } + + System.out.println("Forge map: \"" + info.name + "\""); + if (!info.description.isEmpty()) { + System.out.println("Description: " + info.description); + } + if (!info.author.isEmpty()) { + System.out.println("Author: " + info.author); + } + if (info.mapId >= 0) { + System.out.println("Base map ID: " + info.mapId); + } + System.out.println("Map variant size: " + mapData.length + " bytes"); + + // generate output filename, same scheme as the screenshots + String baseName = inputFile.getName(); + int dotIndex = baseName.lastIndexOf('.'); + if (dotIndex > 0) { + baseName = baseName.substring(0, dotIndex); + } + String outputFileName = baseName + "-reclaimed.map"; + File outputFile = new File(outputDir, outputFileName); + + writeFile(outputFile, mapData); + System.out.println("Saved as: " + outputFile.getAbsolutePath()); + } + + /* + * Reads the entire contents of a file into a byte array. + / + private byte[] readFile(File file) throws Exception { + long length = file.length(); + if (length > MAX_INPUT_SIZE) { + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": file too big for a Forge map"); + } + + byte[] data = new byte[(int) length]; + try (DataInputStream in = new DataInputStream(new FileInputStream(file))) { + in.readFully(data); + } catch (IOException e) { + throw ErrorHandler.makeException(ErrorHandler.BAD_INPUT + ": " + e.getMessage()); + } + return data; + } + + private void writeFile(File file, byte[] data) throws Exception { + try (FileOutputStream out = new FileOutputStream(file)) { + out.write(data); + } catch (IOException e) { + throw ErrorHandler.makeException(ErrorHandler.BAD_OUTPUT + ": " + e.getMessage()); + } + } + + static boolean isSTFSFile(byte[] data) { + if (data.length < 4) return false; + int magic = readInt32(data, 0); + return magic == STFS_MAGIC_CON \|\| magic == STFS_MAGIC_LIVE \|\| magic == STFS_MAGIC_PIRS; + } + + static boolean isBLFFile(byte[] data) { + return data.length >= BLF_CHUNK_HEADER_SIZE && readInt32(data, 0) == BLF_START; + } + + /* + * Lists the files inside a Halo 3 package. The Arbiter uses this to tell + * screenshots (screen.shot) from maps (sandbox.map) without trusting file names. + * + * @throws Exception if the package is damaged or belongs to another game + / + static List listPackageFiles(byte[] data) throws Exception { + StfsPackage stfs = new StfsPackage(data); + if (stfs.titleId != HALO3_TITLE_ID) { + System.out.printf("Package belongs to title %08X, not Halo 3%n", stfs.titleId); + throw ErrorHandler.makeException(ErrorHandler.UNKNOWN_FILE + ": not a Halo 3 package"); + } + List names = new ArrayList(); + for (StfsEntry entry : stfs.entries) { + if (!entry.isDirectory) { + names.add(entry.name); + } + } + return names; + } + + /* + * Saved game file type of an already extracted BLF file + * (FILE_TYPE_SCREENSHOT, FILE_TYPE_MAP_VARIANT...), or -1 if it can't be told. + / + static int readBlfFileType(byte[] blf) { + if (!isBLFFile(blf)) return -1; + int[] chunks = findBlfChunks(blf); + if (chunks == null) return -1; + if (hasMetadata(blf, chunks[0])) { + return readInt32(blf, chunks[0] + METADATA_OFFSET + METADATA_FILE_TYPE); + } + return chunks[1] >= 0 ? FILE_TYPE_MAP_VARIANT : -1; + } + + /* + * Finds sandbox.map in the package and returns its contents. + / + private byte[] extractFromSTFS(byte[] data) throws Exception { + StfsPackage stfs = new StfsPackage(data); + + System.out.printf("Package: \"%s\", title ID %08X%n", stfs.displayName, stfs.titleId); + if (stfs.titleId != HALO3_TITLE_ID) { + throw ErrorHandler.makeException(ErrorHandler.UNKNOWN_FILE + ": not a Halo 3 package"); + } + + StfsEntry mapEntry = null; + for (StfsEntry entry : stfs.entries) { + System.out.println("Package contains: " + entry.name + + (entry.isDirectory ? " (folder)" : " (" + entry.size + " bytes)")); + if (mapEntry == null && !entry.isDirectory && entry.name.equalsIgnoreCase(MAP_VARIANT_FILE)) { + mapEntry = entry; + } + } + + if (mapEntry == null) { + // the "Package contains" lines above tell what is in there instead + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": no Forge map inside"); + } + + byte[] mapData = stfs.readFile(mapEntry); + + if (stfs.badBlocks > 0) { + System.out.println("Warning: " + stfs.badBlocks + " block(s) of " + MAP_VARIANT_FILE + + " do not match the package hashes, the package may be damaged"); + } else { + System.out.println("Package hashes OK"); + } + return mapData; + } + + /* + * Walks the BLF chunks of a map variant, makes sure it really is a Forge map + * and reads its name, description and author. + / + private MapInfo readMapInfo(byte[] blf) throws Exception { + if (!isBLFFile(blf)) { + System.out.println(MAP_VARIANT_FILE + " is not a map variant (BLF) file"); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": not a Forge map"); + } + + int[] chunks = findBlfChunks(blf); + if (chunks == null) { + System.out.println("Map variant has a broken chunk"); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": damaged map variant"); + } + int contentHeader = chunks[0]; + int mapVariant = chunks[1]; + if (chunks[2] < 0) { + System.out.println("Map variant is incomplete, its end of file chunk is missing"); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": damaged map variant"); + } + + // both chdr and mvar start with the same metadata block, chdr is the official one + int metadata = -1; + if (hasMetadata(blf, contentHeader)) { + metadata = contentHeader + METADATA_OFFSET; + } else if (hasMetadata(blf, mapVariant)) { + metadata = mapVariant + METADATA_OFFSET; + } + + int fileType = metadata >= 0 ? readInt32(blf, metadata + METADATA_FILE_TYPE) : -1; + if (mapVariant < 0 && fileType != FILE_TYPE_MAP_VARIANT) { + if (fileType >= 0 && fileType < FILE_TYPE_NAMES.length) { + System.out.println("This file is a " + FILE_TYPE_NAMES[fileType]); + } + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": not a Forge map"); + } + + MapInfo info = new MapInfo(); + if (metadata >= 0) { + info.name = readUtf16(blf, metadata + METADATA_NAME, 32); + info.description = readUtf8(blf, metadata + METADATA_DESCRIPTION, 128); + info.author = readUtf8(blf, metadata + METADATA_AUTHOR, 16); + info.mapId = readInt32(blf, metadata + METADATA_MAP_ID); + } + return info; + } + + /* + * Walks the BLF chunks (4 byte signature, 4 byte size including the header, + * 2+2 byte version) and returns the offsets of the chdr chunk, the map variant + * chunk and the _eof chunk, -1 for the missing ones. Null if a chunk is broken. + / + private static int[] findBlfChunks(byte[] blf) { + int contentHeader = -1; + int mapVariant = -1; + int end = -1; + + int offset = 0; + while (offset + BLF_CHUNK_HEADER_SIZE <= blf.length) { + int signature = readInt32(blf, offset); + int size = readInt32(blf, offset + 4); + if (size < BLF_CHUNK_HEADER_SIZE \|\| size > blf.length - offset) { + return null; + } + + if (signature == BLF_CONTENT_HEADER && contentHeader < 0) { + contentHeader = offset; + } else if ((signature == BLF_MAP_VARIANT \|\| signature == BLF_MAP_VARIANT_PC) && mapVariant < 0) { + mapVariant = offset; + } else if (signature == BLF_END) { + end = offset; + break; + } + offset += size; + } + return new int[] { contentHeader, mapVariant, end }; + } + + /* + * True if there is a chunk at chunkOffset big enough to hold the metadata block. + / + private static boolean hasMetadata(byte[] blf, int chunkOffset) { + if (chunkOffset < 0) return false; + int chunkSize = readInt32(blf, chunkOffset + 4); + return chunkSize >= METADATA_OFFSET + METADATA_MAP_ID + 4; + } + + /* + * What the map variant says about itself. + / + private static final class MapInfo { + String name = ""; + String description = ""; + String author = ""; + int mapId = -1; + } + + /* + * One entry of the STFS file table. + / + private static final class StfsEntry { + final String name; + final boolean isDirectory; + final boolean consecutive; // blocks follow each other, no need for the hash chain + final int blockCount; + final int startBlock; + final int size; + + StfsEntry(byte[] data, int offset) { + int flags = data[offset + 0x28] & 0xFF; + int nameLength = Math.min(flags & 0x3F, 0x28); + int end = offset; + while (end < offset + nameLength && data[end] != 0) { + end++; + } + name = new String(data, offset, end - offset, StandardCharsets.ISO_8859_1); + isDirectory = (flags & 0x80) != 0; + consecutive = (flags & 0x40) != 0; + blockCount = readUInt24LE(data, offset + 0x29); + startBlock = readUInt24LE(data, offset + 0x2F); + size = readInt32(data, offset + 0x34); + } + } + + /* + * Read only STFS reader, just enough to list a package and read files from it. + * + * Data blocks are 4 KB. Every 170 data blocks there is a hash table + * (and above 170 blocks also a level 1 table). Packages made by the console + * keep two copies of each table and a status bit says which one is current. + * Block math follows Free60 and Velocity (hetelek/Velocity). + / + private static final class StfsPackage { + final byte[] data; + final int titleId; + final String displayName; + final List entries = new ArrayList(); + int badBlocks = 0; + + private final int blockSeparation; + private final int tableShift; // 1 = two copies of every hash table, 0 = one copy + private final int allocatedBlocks; + private final long firstHashTable; + private final int topLevel; + private final long topTable; + + StfsPackage(byte[] data) throws Exception { + this.data = data; + + if (data.length < 0x3AD) { + throw damaged("header is cut off"); + } + int headerSize = readInt32(data, 0x340); + if (headerSize < 0x3AD \|\| headerSize > data.length) { + throw damaged("bad header size"); + } + titleId = readInt32(data, 0x360); + displayName = readUtf16(data, 0x411, 0x80); + + // volume descriptor at 0x379, file system type at 0x3A9 (0 = STFS) + if ((data[0x379] & 0xFF) != 0x24 \|\| readInt32(data, 0x3A9) != 0) { + System.out.println("Package does not use the STFS file system"); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": unsupported package type"); + } + blockSeparation = data[0x37B] & 0xFF; + int fileTableBlockCount = readUInt16LE(data, 0x37C); + int fileTableBlock = readUInt24LE(data, 0x37E); + allocatedBlocks = readInt32(data, 0x395); + + tableShift = (~blockSeparation) & 1; + firstHashTable = (headerSize + 0xFFFL) & ~0xFFFL; + + if (allocatedBlocks <= 0) { + throw damaged("no data blocks"); + } else if (allocatedBlocks <= STFS_BLOCKS_PER_TABLE) { + topLevel = 0; + } else if (allocatedBlocks <= STFS_BLOCKS_PER_LEVEL1) { + topLevel = 1; + } else { + System.out.println("Package has " + allocatedBlocks + " blocks, far more than a Forge map needs"); + throw ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": file too big for a Forge map"); + } + + // the top table is the level 0 table for small packages, the level 1 table otherwise + long topTableBlock = (topLevel == 0) ? 0 : levelOneTableBlock(); + topTable = firstHashTable + (topTableBlock << 12) + secondCopyOffset((blockSeparation & 2) != 0); + + readFileTable(fileTableBlock, fileTableBlockCount); + } + + /* + * Reads all file entries, following the block chain of the file table. + / + private void readFileTable(int block, int blockCount) throws Exception { + for (int i = 0; i < blockCount; i++) { + long base = blockOffset(checkBlock(block)); + requireBytes(base, STFS_BLOCK_SIZE); + + for (int slot = 0; slot < STFS_BLOCK_SIZE / STFS_FILE_ENTRY_SIZE; slot++) { + int entry = (int) base + slot STFS_FILE_ENTRY_SIZE; + int flags = data[entry + 0x28] & 0xFF; + if ((flags & 0x3F) == 0) continue; // empty slot + if (data[entry] == 0) break; // end of the listing + entries.add(new StfsEntry(data, entry)); + } + + if (i + 1 < blockCount) { + block = nextBlock(block); + } + } + } + + /** + * Returns the contents of a file in the package. + / + byte[] readFile(StfsEntry entry) throws Exception { + if (entry.isDirectory \|\| entry.size < 0 + \|\| (long) entry.size > (long) entry.blockCount STFS_BLOCK_SIZE) { + throw damaged("bad file table entry"); + } + + byte[] out = new byte[entry.size]; + int block = entry.startBlock; + int copied = 0; + for (int i = 0; copied < entry.size; i++) { + if (i >= entry.blockCount) { + throw damaged("block chain is too short"); + } + long offset = blockOffset(checkBlock(block)); + int length = Math.min(STFS_BLOCK_SIZE, entry.size - copied); + requireBytes(offset, length); + System.arraycopy(data, (int) offset, out, copied, length); + verifyBlock(block, offset); + + copied += length; + if (copied < entry.size) { + block = entry.consecutive ? block + 1 : nextBlock(block); + } + } + return out; + } + + /** + * File offset of a data block: skips the header and every hash table in front of it. + * Valid below 0x70E4 blocks, bigger packages are rejected in the constructor. + / + private long blockOffset(int block) { + long backing = block + (((long) block + STFS_BLOCKS_PER_TABLE) / STFS_BLOCKS_PER_TABLE << tableShift); + if (block >= STFS_BLOCKS_PER_TABLE) { + backing += 1L << tableShift; // the level 1 table(s) + } + return firstHashTable + (backing << 12); + } + + /* + * Block number (counting hash tables) of the level 0 hash table covering a data block. + / + private long levelZeroTableBlock(int block) { + if (block < STFS_BLOCKS_PER_TABLE) return 0; + long groupSize = STFS_BLOCKS_PER_TABLE + (1L << tableShift); + return (block / STFS_BLOCKS_PER_TABLE) groupSize + (1L << tableShift); + } + + /** + * Block number (counting hash tables) of the level 1 hash table. + / + private long levelOneTableBlock() { + return STFS_BLOCKS_PER_TABLE + (1L << tableShift); + } + + private long secondCopyOffset(boolean useSecondCopy) { + return (tableShift == 1 && useSecondCopy) ? STFS_BLOCK_SIZE : 0; + } + + /* + * File offset of the hash entry that describes a data block. + / + private long hashEntryOffset(int block) throws Exception { + boolean secondCopy; + if (topLevel == 0) { + secondCopy = (blockSeparation & 2) != 0; + } else { + // the level 1 entry of this group says which level 0 copy is current + secondCopy = (readByte(topTable + (long) (block / STFS_BLOCKS_PER_TABLE) STFS_HASH_ENTRY_SIZE + 0x14) & 0x40) != 0; + } + return firstHashTable + (levelZeroTableBlock(block) << 12) + secondCopyOffset(secondCopy) + + (long) (block % STFS_BLOCKS_PER_TABLE) * STFS_HASH_ENTRY_SIZE; + } + + /** + * Next block of a chain, as written in the hash entry (24 bit, big endian). + / + private int nextBlock(int block) throws Exception { + long entry = hashEntryOffset(block); + return (readByte(entry + 0x15) << 16) \| (readByte(entry + 0x16) << 8) \| readByte(entry + 0x17); + } + + /* + * Compares the SHA-1 of a data block with the one stored in the hash table. + / + private void verifyBlock(int block, long offset) throws Exception { + long entry = hashEntryOffset(block); + if (offset + STFS_BLOCK_SIZE > data.length \|\| entry + 0x14 > data.length) { + badBlocks++; + return; + } + MessageDigest sha1 = MessageDigest.getInstance("SHA-1"); + sha1.update(data, (int) offset, STFS_BLOCK_SIZE); + byte[] expected = Arrays.copyOfRange(data, (int) entry, (int) entry + 0x14); + if (!Arrays.equals(sha1.digest(), expected)) { + badBlocks++; + } + } + + private int checkBlock(int block) throws Exception { + if (block < 0 \|\| block >= allocatedBlocks) { + throw damaged("block chain points outside the package"); + } + return block; + } + + private void requireBytes(long offset, int length) throws Exception { + if (offset < 0 \|\| offset + length > data.length) { + throw damaged("file is cut off"); + } + } + + private int readByte(long offset) throws Exception { + requireBytes(offset, 1); + return data[(int) offset] & 0xFF; + } + + // short message for the status bar, the reason goes to the console + private static Exception damaged(String reason) { + System.out.println("Damaged package: " + reason); + return ErrorHandler.makeException(ErrorHandler.CANT_PROCESS + ": damaged package"); + } + } + + // helpers, Xbox 360 data is big endian except a few STFS fields + + private static int readInt32(byte[] data, int offset) { + return ((data[offset] & 0xFF) << 24) \| ((data[offset + 1] & 0xFF) << 16) + \| ((data[offset + 2] & 0xFF) << 8) \| (data[offset + 3] & 0xFF); + } + + private static int readUInt16LE(byte[] data, int offset) { + return (data[offset] & 0xFF) \| ((data[offset + 1] & 0xFF) << 8); + } + + private static int readUInt24LE(byte[] data, int offset) { + return (data[offset] & 0xFF) \| ((data[offset + 1] & 0xFF) << 8) \| ((data[offset + 2] & 0xFF) << 16); + } + + /* + * Reads a zero terminated UTF-16 (big endian) string of at most maxBytes bytes. + / + private static String readUtf16(byte[] data, int offset, int maxBytes) { + if (offset >= data.length) return ""; + int limit = Math.min(data.length, offset + maxBytes); + int end = offset; + while (end + 1 < limit && (data[end] != 0 \|\| data[end + 1] != 0)) { + end += 2; + } + return clean(new String(data, offset, end - offset, StandardCharsets.UTF_16BE)); + } + + /* + * Reads a zero terminated UTF-8 string of at most maxBytes bytes. + / + private static String readUtf8(byte[] data, int offset, int maxBytes) { + if (offset >= data.length) return ""; + int limit = Math.min(data.length, offset + maxBytes); + int end = offset; + while (end < limit && data[end] != 0) { + end++; + } + return clean(new String(data, offset, end - offset, StandardCharsets.UTF_8)); + } + + // drop control characters so a damaged name can't mess up the output + private static String clean(String text) { + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < text.length(); i++) { + char c = text.charAt(i); + sb.append(Character.isISOControl(c) ? ' ' : c); + } + return sb.toString().trim(); } -} \ No newline at end of file +} diff --git a/src/TerminalUI.java b/src/TerminalUI.java index c6c2205..015adad 100644 --- a/src/TerminalUI.java +++ b/src/TerminalUI.java @@ -1,7 +1,7 @@ / * Halo 3 Reclaimer [TerminalUI.java] * player data extractor from Xbox 360 version of Halo 3 -* copyright 2025, David Polakovic, licensed under GPLv3 +* copyright 2025, 2026, David Polakovic, licensed under GPLv3 * contact: email@dpolakovic.space / @@ -12,7 +12,7 @@ import java.io.File; * This class contains all the command-line interface logic. * -* @version 1.0 +* @version 1.1 */ public class TerminalUI { @@ -85,26 +85,14 @@ public class TerminalUI { } try { - System.out.println("Processing file: " + inputFile.getName()); - - String fileType = Arbiter.getFileTypeDescription(inputFile); - if (!fileType.equals("Unrecognized file")) { - System.out.println("Detected as: " + fileType); - } - - // use Arbiter to process the file - Arbiter.processFile(inputFile, outputDir); - - String baseName = inputFile.getName(); - int dotIndex = baseName.lastIndexOf('.'); - if (dotIndex > 0) { - baseName = baseName.substring(0, dotIndex); - } - String outputFileName = baseName + "-reclaimed.jpg"; - File outputFile = new File(outputDir, outputFileName); - - System.out.println("File successfully processed to: " + outputFile.getAbsolutePath()); - + System.out.println((inputFile.isDirectory() ? "Processing folder: " : "Processing file: ") + + inputFile.getName()); + + // use Arbiter to process the file or folder + String result = Arbiter.processFile(inputFile, outputDir); + + System.out.println(result + " into: " + outputDir.getAbsolutePath()); + } catch (Exception e) { // parse error code from exception message if it starts with a number String message = e.getMessage(); @@ -124,4 +112,4 @@ public class TerminalUI { public static void showTooManyArguments() { ErrorHandler.exitWithError(ErrorHandler.TOO_MANY_ARGS); } -} \ No newline at end of file +} diff --git a/src/mc.png b/src/mc.png index a9e4591..16d4afc 100644 Binary files a/src/mc.png and b/src/mc.png differ

Field Summary

Field Detail

SCREENSHOTS_DIR

MAPS_DIR

processFile

getFileTypeDescription

Class Cortana

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

Cortana

Method Detail

findStorage

copyHalo3Files

deleteCopies

Uses of ClassCortana

A

D

F

G

P

Classes

Uses of Class
Cortana